Finite State Markov Decision Processes with Transfer Entropy Costs

We consider a mathematical framework of finite state Markov Decision Processes (MDPs) in which a weighted sum of the classical state-dependent cost and the transfer entropy from the state random process to the control random process is minimized. Physical interpretations of the considered MDPs are provided in the context of networked control systems theory and non-equilibrium thermodynamics. Based on the dynamic programming principle, we derive an optimality condition comprised of a Kolmogorov forward equation and a Bellman backward equation. As the main contribution, we propose an iterative forward-backward computational procedure similar to the Arimoto-Blahut algorithm to synthesize the optimal policy numerically. Convergence of the algorithm is established. The proposed algorithm is applied to an information-constrained navigation problem over a maze, whereby we study how the price of information alters the optimal decision polices qualitatively.